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- The MAILING DATE of this communication appears on the cover she t with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )M Responsive to communication(s) filed on 12 November 2003 . 
2a)Q This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-37 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 1-37 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
11 )□ The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)QAII b)Q Some*c)n None of: 

1 0 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1 ) Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) Paper No(s). 



2) □ Notice of Drattsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) Q Other: 
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DETAILED ACTION 



1 . This action is in response to the amendment filed 1 



1/12/2003. 



2. Claims 1-37 have been examined and are pending 



in the application. 



Claim Rejections - 35 USC § 



103 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-15 and 19-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Hashimoto U.S Patent No. 6,1 19,147. 

As to claim 1, Hashimoto teaches a system comprising a multi-modal 
application (1 A, Fig. 56) comprising at least a first and a second mode process (inputs 
and outputs from the application programs, lines 23-25 column 41; speech synthesis 
unit operated as an independent process, lines 37-38 column 42); a multi-modal shell 
(11, Fig. 56) for managing and synchronizing information exchanges between the 
processes (controls by exchanging the messages transmitted from the speech unit to 
the application program, lines 38-41 column 10); the processes register their respective 
active commands and corresponding actions (register the recognition vocabularies and 
the appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 1 1 ) with the multi model shell (1 1 , Fig. 56). Hashimoto does not explitcitly teach 
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an API. However, Hashimoto teaches that each application program includes a 
message I/O unit (line 9-29 column 12) wherein all of the application program 
interactions with the speech recognition system are handled by this message I/O unit. 
Therefore one of ordinary skill in the art would conclude that this message I/O unit could 
be used as an API since it provides the interface for each application program to 
interact with the speech recognition system. 

As to claim 2, Hashimoto as modified further teaches a registry having a 
registration table (program management table, line 58 column 10), managed by the 
multi-modal shell (11, Fig. 56), that comprises a list of each of the registered commands 
and corresponding synchronized actions (the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 1 1) that results in both the first and second mode processes upon execution of 
a registered command (once the speech input "Finish" is transmitted, both of the 
application program can be finished by this single speech input, lines 5-8 column 27) by 
one of the first and second mode processes (inputs and outputs from the application 
programs, lines 23-25 column 41). 

As to claim 3, Hashimoto as modified further teaches a first mono-mode 
application (speech synthesis unit, line 37 column 42) and a second mono-mode 
application (application program, lines 41-42 column 42), wherein the multi-modal shell 
(11, Fig. 56) manages and synchronizes information exchanges (message exchanges 
using the process communication, lines 39-40 column 42) between the first (speech 



Application/Control Number: 09/507,526 Page 4 

Art Unit: 2126 

synthesis unit, line 37 column 42) and second mono-mode applications (application 
program, lines 41-42 column 42). 

As to claim 4, Hashimoto as modified does not teach devices having user 
interface modalities. However, Hashimoto teaches (lines 51-55 column 12) the 
message system can be implemented as a server and clients system wherein the 
speech recognition unit can act as a server and the applications programs are clients. 
Therefore one of ordinary skill in the art would conclude that the clients are the devices 
with modalities wherein these devices can register their commands and corresponding 
actions with the server. 

As to claim 5, Hashimoto as modified further teaches the devices multi-modal 
shell are distributed over a network (server and client system, lines 51-55 column 12), 
and wherein the API is implemented using distributed APIs or protocols (byte stream 
type protocol, lines 54-55 column 12). 

As to claim 6, Hashimoto as modified further teaches a mechanism for 
converting a mono-mode application to a multi modal application (application program 5 
is interact with both speech input and keyboard input, Fig. 17). 

As to claim 7, Hashimoto as modified further teaches the mono-mode 
application is a GUI application (user interface of Fig. 18), the mechanism (11, Fig. 56) 
provides speech enablement (controls by exchanging the messages transmitted from 
the speech unit to the application program, lines 38-41 column 10) of the GUI 
application (user interface of Fig. 18) by registering the active commands of the GUI 
application and building a grammar for the registered commands to support the 
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commands in a speech modality (register the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 11). 

As to claim 8, Hashimoto as modified further teaches a mechanism for building 
a multi-modal application (application program 5 is interact with both speech input and 
keyboard input, Fig. 17). 

As to claim 9, Hashimoto as modified further teaches the mechanism (11, Fig. 
56) is used for directly programming the registry by building a registration table 
(program management table, line 58 column 10) having user-defined commands and 
corresponding actions (the recognition vocabularies and the appropriate actions that 
response to these vocabularies, line 48 column 10 to line 10 column 11) for each of the 
modalities of the multi-modal application (1A, Fig. 56). 

As to claim 10, Hashimoto as modified further teaches that the system is 
implemented on personal computers, workstations... (lines 8-12 column 1). Hashimoto 
does not explicitly disclose an operating system. "Official Notice" is taken that both the 
concept and advantage of providing for an operating system is well known and 
expected in the art. It would have been obvious to include an operating system into the 
system of Hashimoto because it would provide the execution space for the system. 

As to claim 11, Hashimoto as modified further teaches the system is distributed 
over a network (server and client system, lines 51-55 column 12). 

As to claim 12, Hashimoto as modified further teaches the multi-modal 
application (1 A, Fig. 56) is a multi-modal browser (a mail browser, Fig. 27), wherein the 
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first mode process renders a first modality (the user can command to open the mail by 
saying "yes", '* nes 54-60 column 25) and the second mode process renders a second 
modality (the mouse can click on "yes" to open the mail, lines 61-65 column 25). 

As to claim 13, Hashimoto as modified further teaches the first modality is GUI 
(interface of Fig. 27) and the second modality is speech (the user can command to open 
the mail by saying "yes", lines 54-60 column 25). 

As to claim 14, Hashimoto as modified further teaches the multi-modal shell (11, 
Fig. 56) processes a CML file (recognition result, line 16 column 25) to send modality 
specific presentation information (speech result from the speech recognition system, 
lines 14-17 column 25) in the CML file (recognition result, line 16 column 25) to the 
respective browsers (a mail browser, Fig. 27). 

As to claim 15, Hashimoto as modified further teaches the CML file (recognition 
result, line 16 column 25) encapsulates the modality-specific presentation information 
(speech result from the speech recognition system, lines 14-17 column 25) in a single 
modality-independent representation (input speech, line 12 column 25). 

As to claim 19, Hashimoto teaches receiving a command (speech input, line 14 
column 41) in a first modality (speech recognition interface system, line 13 column 41); 
triggering an action (speech output, line 19 column 41) in the first modality (speech 
recognition interface system, line 13 column 41) and a corresponding action (the mail is 
opened when the user saying "yes", lines 54-60 column 25) in at least a second 
modality (application program, lines 41-42 column 42) based on the received command; 
updating application state associated with the second modality (update the program 
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management data in the program management according to an internal state of each 
application program, lines 35-39 column 76). Hashimoto does not explitcitly teach 
updating application state associated with the first modality. However, Hashimoto 
teaches (lines 37-46 column 14) the speech recognition system also has its own internal 
state; and when there is a change in this internal state, the application program will get 
a message notifies about this change. Therefore one of ordinary skill in the art would 
conclude that in a certain time, the internal state of the speech recognition system is 
updated; therefore produces the state change. 

As to claim 20, Hashimoto as modified further teaches registering active 
commands associated with the modalities (the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 11); associating, with each registered command of the modalities, an action on 
one modality and a corresponding action on the other modality (once the speech input 
"Finish" is transmitted, both of the application program can be finished by this single 
speech input, lines 5-8 column 27). 

As to claim 21, Hashimoto as modified further teaches building a command to 
action registration table (program management table, line 58 column 10) based on the 
registered commands and actions (the recognition vocabularies and the appropriate 
actions that response to these vocabularies, line 48 column 10 to line 10 column 11). 

As to claim 22, Hashimoto as modified further teaches the registration table 
(program management table, line 58 column 10) is built by a multi-modal shell (1 1 , Fig. 
56). Hashimoto does not explitcitly teach an API. However, Hashimoto teaches that 
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each application program includes a message I/O unit (line 9-29 column 12) wherein all 
of the application program interactions with the speech recognition system are handled 
by this message I/O unit. Therefore one of ordinary skill in the art would conclude that 
this message I/O unit could be used as an API since it provides the interface for each 
application program to interact with the speech recognition system. 

As to claim 23, Hashimoto as modified further teaches looking up the received 
command (the recognition vocabulary lists, lines 59-60 column 10) in the registration 
table (program management table, line 58 column 10); and executing the actions 
associated with the received command on the first and second modalities (once the 
speech input "Finish" is transmitted, both of the application program can be finished by 
this single speech input, lines 5-8 column 27). 

As to claim 24, Hashimoto as modified further teaches registering a callback 
handle for each of the registered commands to notify the first and second modalities of 
completion of the actions corresponding to the registered commands (after the 
dictionary production is completed, notifies this act to the data acquisition unit 8 by a 
message indicating the completion of the dictionary production, lines 3-5 column 33). 

As to claim 25, Hashimoto as modified further teaches executing the callback 
handle associated with the received command to trigger callback actions on the 
modalities (after the dictionary production is completed, notifies this act to the data 
acquisition unit 8 by a message indicating the completion of the dictionary production, 
lines 3-5 column 33). 
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As to claim 26, Hashimoto as modified further teaches executing first thread 
associated with the received command ("Finish", line 6 column 27); triggering a 
corresponding second thread to initiate the corresponding action on the second 
modality (once the speech input "Finish" is transmitted, both of the application program 
can be finished by this single speech input, lines 5-8 column 27). 

As to claim 27, Hashimoto as modified further teaches the threads are applets 
("Yes" and "No" icons in Fig. 27). 

As to claim 28, Hashimoto as modified further teaches the threads communicate 
via socket connections (server and client system (lines 51-55 column 12) with a mail 
browser, Fig. 27)). 

As to claims 29-37, they are device claims of claims 19-21 and 23-28, 
respectively. Therefore, they are rejected for the same reasons as claims 19-21 and 
23-28 above. 

4. Claims 16-18 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hashimoto in view of Toomey U.S Patent No. 6,119,147. 

As to claim 16, Hashimoto as modified does not explicitly teach the file 
comprises a combination of declarative markup languages. Toomey teaches a multi- 
model document with a combination of declarative markup languages (text discussion, 
audio commands, graphics, and documents, lines 60-61 column 1 ). It would have been 
obvious to apply the teachings of Toomey to the system of Hashimoto because this 
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document will provide the user with the convenience of interacting with the system using 
the choice of texts or speech commands. 

As to claim 17, Toomey further discloses a single file combining the declarative 
markup languages and synchronization elements to provide tight synchronization 
between the declarative markup languages (interactions are inserted into the multi- 
modal document at a point that is chronological in the meeting to create a synchronous 
meeting, lines 1 1-13 column 15). 

As to claim 18, Toomey further discloses a separate file for each of the 
declarative markup languages (multiple tracks in the multi-modal document, lines 59-60 
column 1). 

Response to Arguments 

5. Applicant's arguments filed have been fully considered but are moot in view of 
the new ground(s) rejection. 

Applicant's arguments presented issues which required the Examiner to further 
view the previous rejection. The Examiner conducted a further search regarding the 
issues mentioned in Applicant's response. Therefore, all arguments regarding the cited 
references of the previous rejection are moot in view of the new grounds of rejection. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to The Thanh Ho whose telephone number is 703-306- 
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5540. A voice mail service is also available for this number. The examiner can 
normally be reached on Monday - Friday, 8:30 am - 5:00 pm. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is 703-305- 
3900. 

Any response to this action should be mailed to: 

Commissioner for Patents 

P.O Box 1450 

Alexandria, VA 22313-1450 
Or fax to: 

• AFTER-FINAL faxes must be signed and sent to (703) 746 - 7238 

• OFFICAL faxes must be signed and sent to (703) 746 - 7239 

• NON OFFICAL faxes should not be signed, please send to (703) 746 - 7240 



TTH 

January 23, 2004 





